Interconnects for Large Computer Systems

This white paper explores issues in extending the Lightfleet direct-broadcast interconnect for addressing large numbers of endpoints. In particular, connectivity issues in cloud computing, server farms, supercomputing, storage farms, and telecommunication installations are discussed. The several approaches presently in use are compared and contrasted to similar methods enabled by the Lightfleet interconnect. The central issue in addressing large systems of endpoints, which may be servers, high-performance computing nodes, storage nodes, gateways, or communication points, is that of scalability: can the interconnect easily be extended to include more nodes? Typical commodity interconnects are based on switches that have a limited number of connection ports, so the issue of (horizontal) scalability becomes that of interconnecting large numbers of port-limited modules. Certain topologies (configurations of nodes) are in common use with the most frequent ones being mesh, star, and tree. Hybrid versions are also in use (a tree of meshes, for example). The first section, following an introduction, discusses issues of network topology, showing that different topologies have different strengths and weaknesses; the trade-offs are usually between scalability, bandwidth between nodes, number of nodes serviced, network diameter (a measure of the path distance between nodes), and reachability or path redundancy (the number available paths between two nodes). Two key parameters are diameter (which determines the latency between nodes) and bandwidth. Trade-offs between diameter and bandwidth or diameter and scalability are assessed. The goal of the first part of this paper is so show how to select a fabric topology based on identical fabric modules. The second section is concerned with fabric management and contrasts the control of packet traffic in a switched system with that possible in packet-flow system that is based on a modification to the Lightfleet interconnect. This new method of packet routing operates across interconnects at all scales, from the small enterprise of a few hundred servers to very large server farms and packet routing systems with millions of nodes. The routing methods suggested avoid problems of scale that are endemic to switched fabrics. The analogy being that of unscheduled and locally controlled automobile traffic compared to railroad or airline traffic that requires global management of the entire system. Introduction Background Lightfleet Corporation has invented, patented, and developed a novel interconnect fabric that provides tight coupling between computing nodes. The basic invention allows up to 128 computing nodes (c-nodes or endpoints) and beyond to be fully interconnected without possibility of in-fabric congestion. That is, all paths between c-nodes are strictly non-blocking and each endpoint may broadcast to all other endpoints simultaneously without the need for software control of path routing. The white paper, Properties of Lightfleet’s Direct Broadcast Optical Interconnect, referred to as DBOI, discusses the properties of such an interconnect fabric, showing that the Lightfleet device is optimal in the eight basic interconnect properties (bandwidth, degree, diameter, latency, capacity, congestion, data skew, and path freedom). Since the DBOI is strictly non-blocking, it has no in-fabric congestion so its behavior under Amdahl’s law of scalability with additive communications overhead is nearly ideal. The white paper DBOI Performance Characteristics discusses additional characteristics of the interconnect fabric, demonstrating both the functional advantage of the interconnect and discussing its competitive advantage even under advancing communications technology. The topics of bit rate, broadcast, and how response-time variance of messages within a DBOI system are improved are all covered. In addition, several programming models are presented. The purpose of the present white paper is to show how the basic DBOI may be used to interconnect thousands to millions of endpoints while retaining many of essential ideal properties that make this such an effective solution for tight clusters. The basic idea is to form a network of DBOI modules, where each DBOI module may serve both as a node in the network fabric (or f-node) while, at the same time, function as broadcast interconnect for a tight cluster of computing nodes or endpoints (c-nodes), logically residing at each f-node. This has certain similarities to the way data centers are presently architectured. Lightfleet Corporation V 2.0 Tue 21 Aug 2012 The purpose of the present white paper is to show how the basic DBOI may be used to interconnect thousands to millions of endpoints while retaining many of essential ideal properties that make this such an effective solution for tight clusters. The basic idea is to form a network of DBOI modules, where each DBOI module may serve both as a node in the network fabric (or f-node) while, at the same time, function as broadcast interconnect for a tight cluster of computing nodes or endpoints (c-nodes), logically residing at each f-node. This has certain similarities to the way data centers are presently architectured. While the broadcast capability is maintained, the simultaneous all-to-all property must be modified and the fabric diameter will increase, meaning that the message latency between two arbitrary c-nodes will increase when messages travel between f-nodes. However, as will be shown below, many desirable properties of the resulting fabric of DBOI f-nodes are retained. The advantages of a network of DBOIs over that of a network of switched interconnects (known as a switched fabric) are discussed as well. A DBOI fabric constructed in this manner may be termed a dataor packet-flow fabric to distinguish it from the ubiquitous switched fabric. “Fabric” or “interconnect fabric” is a common term loosely used to refer to a wide range of hardware configurations for interconnecting large numbers of computing nodes and/or storage nodes. As a technical term, “fabric” has become an easy-to-use term often synonymous with “interconnect.”